Method and apparatus for achieving panchromatic response from a color-mosaic imager

ABSTRACT

A method and apparatus for achieving monochromatic response from a low-cost color imager is presented. In this method and apparatus, the out-of-band response to infrared (IR) light is by solid state sensors exploited to produce a monochrome image. The monochrome image is produced by removing the IR blocking filter from the sensor and illuminating the scene to be imaged with IR radiation from an LED. The wavelength emitted from the LED is matched to the wavelength or wavelengths that correspond to a region where the sensor&#39;s response to IR light is relatively even, despite the color-mosaic filter permanently attached to the sensor.

FIELD OF THE INVENTION

This invention relates to imaging systems, particularly those applied tomachine vision systems. In particular it relates to use of color imagersin machine vision systems. More particularly it relates to the use ofcolor imagers to create monochrome images for machine vision processing.

BACKGROUND OF THE INVENTION

Traditionally, the market for focal-plane array solid state imagers,referred to in the art by their underlying technologies (i.e. CCDs forCharge Coupled Devices or CMOS for Complimentary Metal OxideSemiconductor) has been driven by the market for video cameras, of whichthe CCD or CMOS sensor is a main component. Video cameras, as used forstudio broadcast, electronic news gathering (ENG), and consumercamcorders (where the video camera and tape deck or disk drive recorderare integrated into a single compact package) all typically provide acolor signal. This signal is typically generated by either of twomethods.

The first method is typically applied to professional grade equipmentand in the higher-grade of consumer equipment. For this method theincoming image passes through a video lens, and then passes through adichroic prism cluster. The dichroic prism cluster is an assembly ofthree prisms, two of which have a dichroic (color-band selecting)coating. The first prism will have a surface which reflects one colorband, typically blue, to a first CCD or similar imager, and passes theremainder of the colors. The second prism will have a surface whichreflects a second color band, typically green, to a second CCD orsimilar imager. A third prism will accept the balance of the signal, inthis case the red band, and transmits it to a third CCD or similarimager. If there is a coating on the third prism, it is typically acoating to reject out-of-band light to which the CCD may be sensitivebut which is considered undesirable, for example infrared light. Thiscoating to reject infrared (IR) light may alternatively be placed at thefront of the prism cluster, or incorporated into a protective window, orincorporated into the lens. Note that in this first method, each of theCCDs or similar sensors are deployed with their full color spectralresponse intact as the color-separation is accomplished in the dichroicprism cluster. Furthermore each of the sensors acquires image data atits full intrinsic resolution, yielding three full resolution imagesrepresenting three colors from the scene imaged.

The second method is typically used in lower grade equipment such aslower cost and/or very compact ENG systems and in most consumer-gradecamcorder systems. Here a single CCD or similar imager is used. Acolor-mosaic filter is integrated with the solid state imager, where byeach picture element (pixel) of the imager array is covered by acorresponding color-band filter. FIG. 1 shows a diagram of acolor-mosaic filter 13 showing exemplary red 12, green 14 and blue 16color filter elements. Typically, one half of the pixels will beassociated with a color-band filter which transmits primarily the greenimage component, one quarter of the pixels will be associated with acolor band filter which transmits primarily the blue image component,and one quarter of the pixels will be associated with a color bandfilter which transmits primarily the red image component. This type offilter is sometimes referred to as a Bayer filter. This system is biasedtoward the green spectrum as a deliberate compromise to best satisfy thehuman observer. Humans are most sensitive to green light and lesssensitive to blue and red light. The green component carries the mostinformation regarding human faces to which humans are particularlysensitive. Note that in this method the color-mosaic filter isinseparable from the solid state imager, since the filter is alignedprecisely with discreet photosensitive elements in the sensor during themanufacturing process and must remain aligned to perform properly. FIG.2 shows a schematic diagram of a prior art solid state video sensor withattached color-mosaic filter, showing the sensor 20 with attachedcolor-mosaic filter 22, optional IR blocking filter 24, optics assembly26 and controller 30 attached to sensor by cable 28.

Secondary industries including machine vision are relatively small usersof video sensors and cameras compared to the broadcast, ENG and consumercamcorder industries. As a result manufacturers typically do notmanufacture sensors specifically for these secondary markets andtherefore these applications must rely on sensors manufactured for otheruses. Machine vision applications often require high speed processing,and high sensitivity, high resolution and low cost sensors to besuccessful. For these reasons, and others, machine vision applicationstypically use monochrome sensors, where image data from the scene imagedis rendered in shades of grey. To achieve monochrome imaging, machinevision applications sometimes use sensors developed for use inthree-channel dichroic prism cluster professional equipment. This yieldshigh speed processing since color information does not need to beprocessed, high sensitivity since the entire sensor is available tosense broad band illumination and high resolution since all of thepicture elements or pixels are used. This is not, however, a low costsolution since these sensors typically cost much more than mass producedcolor mosaic sensors.

Color mosaic sensors are desirable to be used for machine visionapplications because of their low cost. As a result of the permanentlyattached color filters, these sensors do not perform as well asmonochrome sensors, typically requiring additional processing toeliminate color information, thereby forming a monochrome image. Sincethese cameras are designed to produce color information, the image dataa from these sensors is typically transmitted either as three separate(red, green and blue) images or encoded in one of the many color formatsavailable. One result of this processing is to reduce resolution, sincespatially separate red, green and blue pixels must be combined to form asingle monochrome pixel, thereby reducing spatial resolution. Inaddition, the color filters reduce each pixels sensitivity, requiringlonger exposure times and/or more light energy on the scene, neither ofwhich is desirable.

The sensitivity of solid state imagers to IR is well known. Methods andapparatus for simultaneously imaging visible and infrared have beendeveloped. For instance, U.S. Pat. No. 3,806,633 Multispectral DataSensor and Display System discusses a dual sensor arrangement withoptics to produce an IR image registered with a visible image. U.S. Pat.No. 4,651,001 Visible Infrared Imaging Device with Stacked CellStructure seeks to produce simultaneous visible and IR images using anovel sensor architecture. The ability of solid state sensors to acquireboth visible and IR data is exploited by U.S. Pat. No. 5,555,464Red/Near-Infrared Filtering for CCD Cameras, wherein a novelmosaic-color filter is used instead of the standard filter to admit IRlight in addition to visible light. These approaches all produce colorimages in addition to IR images or produce false color images of IR dataor IR and visible data combined, all of which tend to work against themachine vision goals of cheap, fast, high resolution processing.

There are now other major markets for CCD, CMOS and similar sensors,such as digital still cameras and cellular mobile phones that use colorsensors of the color-mosaic filter type. These developments have led tothe availability of sensors of much reduced cost that might be appliedto machine vision systems and thereby reduce the cost of such systems.One system for using these low cost sensors to image IR data is shown inU.S. Pat. No. 7,235,775 Infrared Imaging Apparatus for ProcessingEmissions in Infrared and Visible Spectrum. The system described thereinremoves the IR filter from the sensor, replacing it with a filter thatpasses only IR data. The system then relies on the visible lightcolor-mosaic filter to form separate three peaks in the IR response andthereby form a false color image of three spectral bands in the IRwavelength region. This approach fails to address two of the majorproblems with using a color-mosaic sensor to acquire IR data. First, thesystem must process three separate images, which is not an improvementover standard visible processing. Secondly, the three false color imagesacquired have reduced resolution with respect to the potentialresolution of the sensor, since they are spatially multiplexed.

Therefore it is evident that color-mosaic sensors have not beenadaptable to the special requirements associated with machine vision dueto the limitations mentioned above. Accordingly, there remains a needfor of the reduced cost sensors that can be applied to machine visionapplications by obtaining a panchromatic response from color-mosaicfiltered solid state imagers.

SUMMARY OF THE INVENTION

The invention is a method of and apparatus for simulating panchromaticresponse from a color-mosaic video sensor. In one embodiment of theinstant invention a set of chromatic response curves depicting responseof the multi-color video sensor as a function of wavelength is examined,identifying a region of said chromatic response curves where themulti-color video sensor has a substantially uniform response to eachwavelength. FIG. 3 is a graph 40 showing the spectral response of atypical color-mosaic filter. This graph shows the response in percentquantum efficiency of the blue 42, green 44 and red 36 filters as afunction of wavelength. Note the uniform response of all three filtersin the IR portion of the spectrum. This wavelength region is in the IRwavelength band, in the wavelengths from about 700 nm to 1000 nm,centered about 850 nm. This wavelength region is typically blocked incommercially available cameras in order to prevent IR radiation fromreaching the sensor and unfavorably affecting the image. The instantinvention removes this IR filter and optionally replaces it with afilter that blocks visible light and passes IR radiation. In the instantinvention, a wavelength or wavelengths are selected from this wavelengthregion and an imaging scene to be imaged by the multi-color video sensoris illuminated with the selected wavelength(s) within the identifiedregion, thereby acquiring IR data from the sensor related to the imagingscene. A schematic diagram of a system constructed according to theinstant invention is shown in FIG. 4 showing the sensor 50 with attachedcolor-mosaic filter 52, optional IR pass filter 54, optical assembly 56,cable 58 connecting the sensor 50 to the controller 60 along with theillumination source 62 with its cable 64 operatively connecting theillumination source 62 to the controller 60.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a prior art image sensor with a mosaicfilter

FIG. 2 is a schematic diagram of a prior art color-mosaic solid statecamera.

FIG. 3 is a graph showing the spectral response of three colors of amosaic filter

FIG. 4 is a schematic diagram of a monochrome machine vision systemusing a color-mosaic sensor

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1 shows standard Bayer tiling for the color mosaic filter 10 thatis aligned with a CCD or CMOS solid state imager. Exemplary red 12,green 14 and blue 16 filter elements are indicated. Proper applicationof this type of filter to a solid state imaging sensor requires that theeach color section of the filter be aligned with a single active pictureelement in the sensor. This is so that each sensor element or pixel willbe exposed to only one filtered color. This requires that the filter becarefully aligned and permanently attached to the sensor. By analysis ofthe diagram, it is apparent that for an image formed with the aid of agreen illuminant the image will be formed with essentially one half ofthe resolution of the equivalent panchromatic sensor, and for blue orred illuminants the resulting image will be one quarter the resolutionof the equivalent panchromatic imager.

For machine vision, it is common practice to control the illuminator foroptimal geometry of the light and for timing of the light (stroboscopicor flash applications). The most practical illuminant for theseapplications is often light emitting diodes (LEDs) lighting which istypically quasi-monochromatic spanning only a few nanometers inwavelength. Although white LEDs are available in at least two differenttechnologies (tri-stimulus and secondary emission types) neither one isavailable in the variety of package types and styles of the moreestablished single-color LEDs. Furthermore, even if suitable broadbandillumination is available, it is often the form of the object and notthe color of the object that is under study, and if color information ispresent (such as by the use of a color imager and a broad-bandilluminant) it may present spurious information that must becomputationally filtered out or ignored.

FIG. 2 shows a typical prior art color-mosaic solid state video systemincluding a solid state sensor 20 with attached color-mosaic filter 22.Light enters the system through the optical assembly 26; passes throughan optional IR filter 24 to block IR light from reaching the sensor, andthen through the color-mosaic filter 22 to the solid state sensor 20.Image data formed by the sensor 20 is transmitted via cable 28 tocontroller 30 for processing. Most but not all color-mosaic imagerscontain an IR filter. The purpose of the IR filter is to filter outextraneous information coming from IR radiation in the scene that woulddistort the color response of the imager. For most commercialapplications, the goal is accurate color fidelity, as opposed to maximumsensitivity or resolution. Since the IR filter does not requirepermanent precise alignment, it is typically either attached to theouter surface of the sensor 20 on top of the color mosaic filter 22, orincluded in the optical assembly. Regardless of where it is found in thesystem it is usually possible to remove the IR filter and replace itwith another type of filter if desired.

FIG. 3 is a graph 40 showing the spectral response of a typicalcolor-mosaic filter. This graph shows the response in percent quantumefficiency of the blue 42, green 44 and red 36 filters as a function ofwavelength. What can be seen by analysis of this image is that all ofthe bands have “leakage” in the infrared region. Typically, this leakageis addressed with an optical filter that rejects this “out-of-band”radiation. However, the instant invention exploits this leakage. In apreferred embodiment, an illuminant is chosen at the point of highestand most nearly matched responsivity. This matched responsivitytypically occurs between 700 nm and 1000 nm. In one embodiment of theinstant invention, illumination at 850 nm will result in substantiallypanchromatic response. It also happens that this color is a common LEDtype and is readily available in a wide variety of packages and sizes.Such devices are typically based on the semiconductor material AluminumGallium Arsenide, which has a band gap that corresponds to emission ator near 850 nm.

FIG. 4 shows a system constructed according to the instant invention. Anembodiment of this invention uses a commercially available color-mosaicimager 50 or a camera based on the same with color-mosaic filter 52attached. Near panchromatic response is achieved with said imager orcamera by removing the IR-blocking filter and optionally replacing itwith a filter 54 that passes IR radiation in the selected wavelengthsand blocks visible light. In a preferred embodiment, aquasi-monochromatic illuminator 62 of a particular wavelength range isused, specifically that wavelength range that corresponds to thespectral response of the imager where all three color channels aresubstantially equal or a subset thereof. In this case the wavelengthrange is between 700 and 1000 nm. In particular a wavelength of about850 nm is used. In a preferred embodiment, a quasi-monochromaticilluminator is selected at the relative peak of responsivity within therange where response to all three color channels is substantially equal.The illuminator 62, under control of and connected to a controller 60via a cable 64 illuminates the scene (not shown) at the appropriate timeand with the selected wavelength or wavelengths. The scene is imaged viathe optical assembly 56 which comprises a lens or lenses operative tofocus IR radiation emitted by illuminator 62 and reflected by scene (notshown) and imaged onto the sensor 50, through the color-mosaic filter 52and the optional IR pass filter 54. The image data thusly acquired bythe sensor 50 is transmitted via cable 58 to controller 60 for furtherprocessing.

In an embodiment of the instant invention, the initial processinginvolves forming a single monochromatic image from three pseudo-colorimages transmitted by the sensor to the controller. Note that colorinformation can be encoded in a variety of ways. The simplest is RGBformat where three separate images with reduced resolution comparable totheir reduced sampling of the sensor area are transmitted. Otherencodings such as YUV or NTSC are possible and would work with thismethod, albeit with possible reduced spatial resolution or dynamicrange. This initial processing creates a monochrome image by filling ina 2D array with dimensions same as the original full size sensor withdata from the three pseudo color images. The data from the “red”, “blue”and “green” images are placed into the monochrome image array accordingto where the particular pixel was acquired from on the original sensorthereby creating a monochrome image with the same intrinsic resolutionas the original sensor. Furthermore, since the scene was illuminatedwith IR light that is selected to be equally transmitted by all threecolors, the sensitivity of the system to light is maximized.

Another embodiment of this invention further processes the data toremove any variances in response by the color-mosaic sensor to thefiltered IR data. It is conceivable that different colors in thecolor-mosaic filter/sensor combination might respond differently to IRradiation. Since the instant invention depends upon having uniformresponse from the sensor regardless of the filter, this difference inresponse can be eliminated by measuring the response over the dynamicrange of the sensor with the wavelength of IR light selected andcalculating correction factors based on the response. These factors canbe additive or multiplicative and are applied to each pixel by thecontroller depending upon which color filter it was acquired through inorder to eliminate or reduce the variance in response between thevarious colors of the color-mosaic filter.

It will be apparent to those of ordinary skill in the art that manychanges may be made to the details of the above-described embodiments ofthis invention without departing from the underlying principles thereof.The scope of the present invention should, therefore, be determined onlyby the following claims.

1. An improved method of simulating panchromatic response from amulti-color video sensor, said multi-color video sensor having achromatic response curve and including a color filter and possibly aninfrared filter, said color video sensor operative to image an imagingscene and acquire image data, said data being transmitted to acontroller, the improvement comprising: identifying a region of saidchromatic response curves where said multi-color video sensor has asubstantially uniform response to each wavelength; illuminating saidimaging scene by said multi-color video sensor with a wavelength withinsaid identified region and; acquiring said data from said multi-colorvideo sensor related to said imaging scene, transmitting said acquireddata to said controller and combining said data into a monochromaticimage.
 2. The method of claim 1, further comprising: determining adifference in response, if any, over said substantially uniform responseregion, and; applying a correction to the data based on said determineddifference.
 3. The method of claim 1 wherein said color filter is aBayer filter having red, green and blue filter components.
 4. The methodof claim 1 where said substantially uniform response to each wavelengthis improved by removing said infrared filter from said color videosensor.
 5. The method of claim 1 where said substantially uniformresponse is improved by adding a visible light filter to said colorvideo sensor,
 6. The method of claim 1 wherein said illuminationincludes wavelengths between about 700 nm to 1000 nm
 7. The method ofclaim 1 wherein said illumination includes a wavelength of about 850 nm.8. An improved system for simulating panchromatic response from amulti-color video sensor, said multi-color video sensor including acolor filter having a chromatic response curve and which may have anassociated infrared filter, said color video sensor operative to imagean imaging scene and acquire image data, and a controller, saidcontroller operative to process said data, the improvement comprising:an illumination source matched to a substantially uniform responseregion of said chromatic response curve.
 9. The system of claim 8wherein said color filter is a Bayer filter having red, green and bluefilter components.
 10. The system of claim 8 where said substantiallyuniform response to each wavelength is improved by removing saidinfrared filter from said color video sensor.
 11. The system of claim 8where said substantially uniform response to each wavelength is improvedby adding a visible light filter to said color video sensor.
 12. Thesystem of claim 8 wherein said illumination source includes wavelengthsbetween about 700 nm to 1000 nm
 13. The system of claim 8 wherein saidillumination source includes wavelengths of about 850 nm.